DOCUMENT RESUME 



ED 455 865 



JC 010 540 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 
ISSN 

PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Grubb, W. Norton 

From Black Box to Pandora's Box: Evaluating 
Remedial/Developmental Education. CCRC Brief Number 11. 
Columbia Univ. , New York, NY. Community Coll. Research 
Center . 

Alfred P. Sloan Foundation, New York, NY. 

ISSN- 1526 -2 04 9 
2001 - 02-00 
6p. 

Reports - Descriptive (141) 

MF01/PC01 Plus Postage. 

College Role; ^Community Colleges; ^Course Evaluation; 
^Curriculum Evaluation; ^Developmental Studies Programs; 

* Evaluation Methods; Evaluation Needs; Remedial Instruction; 
Two Year Colleges; Validated Programs 



ABSTRACT 



The roster of developmental programs available to students 
in community colleges is expanding, though remedial education itself does 
remain a marginal aspect of higher education. The programs are usually 
under-f unded, segregated from "regular" offerings, and taught by part-timers. 
This report suggests that the colleges are sending signals that developmental 
education is not "real" education. The author details the vast array of 
different approaches to developmental /remedial education available in 
institutions of higher education. These approaches range from Shaughnessy ' s 
student- centered approach to the more conventional "skills-and-drills " 
approach and the remedial education offered in the welfare- to-work programs. 
The author finds that most educators who have made the transition form x 
didactic to constructivist teaching have done so on their own, and thus the 
teaching methods seem random and idiosyncratic. The author recommends an 
eclectic approach to the evaluation of remedial education, which includes six 
parts: (1) investigation of the drop-out rates in remedial education; (2) 

outcome measures that include more than test scores of basic skills; (3) use 
of comparison or control groups in order to better evaluate the impact of 
completion; (4) understanding of the program being evaluated; (5) comparisons 
of different approaches to teaching; and (6) better understanding of the 
"assignment" problem. The author concludes that a program of evaluation and 
improvement is central to improving the performances of developmental and 
remedial students. (NB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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W. Norton Crubb 

Remedial/developmental education has burgeoned 
over the past decades, in a variety of postsecondary 
institutions. Unfortunately, while debates for and 
against have been vociferous, the effectiveness of 
these programs has not been as visible an issue. 
Relatively few evaluations of remedial programs have 
been conducted, and many existing evaluations are 
useless because, failing to recognize what the program 
does, they provide little information about what should 
be changed to make it more effective. In place of this 
kind of “black box” evaluation, I recommend a variety 
of evaluation approaches that can improve information 
about many different aspects of remediation, including 
not only its effects but also the instructional methods 
used, the progress of students, and the ways students 
are assigned to remedial programs. I call this a 
“Pandora's Box” approach because it is designed to 
open up the black box, to reveal the problems with 
existing programs including the potential reasons for 
their effectiveness or ineffectiveness — and then to 
improve them. 

Remedial/Developmental Education 

There is almost no discussion about what remedial 
education looks like — what goes on in classrooms, 
whether it is educative in any sense of the word, 
whether it stands any chance of bringing students up 
to “college level.” Without understanding the variety of 
activities that march under the banner of remedial 
education, it is easy to assume that developmental 
education is well defined and can be readily evaluated 
like any other program. Developmental education takes 
place in such a variety of settings — adult basic ed, job 
training, welfare-to-work, community colleges as well 
as four-year colleges — that it is difficult to characterize 
what happens. 

This Brief focuses on remedial/developmental 
education in community colleges. Estimates of the 
proportion of students needing remediation varies from 



25 percent to 50 percent to 78 percent in Tennessee 
(Grubb and Kalman, 1994). Of all the postsecondary 
institutions that offer remedial education, community 
colleges have the greatest chance of doing it well. 
Unlike adult education and job training, with their 
reliance on untrained instructors hired in casual ways 
for part-time work, instructors in community colleges 
generally have master's degrees and are part of 
colleges that consider themselves “teaching 
institutions.” Although these colleges rely too much on 
part-time instructors, there is still a commitment to 
teaching as a career. 

In examining numerous colleges, several distinct 
approaches to developmental education are apparent 
(Grubb and Associates 1999, Ch. 5): 

Skills and drills. By far the most common 
approach to developmental education within 
community colleges is the approach I have labeled 
“skills and drills." This tends to focus on arithmetic 
procedures, on grammar, punctuation and vocabulary, 
on math “problems" of the most contrived sort, and 
passages from texts that have been simplified for low 
reading levels. Because of the common pattern of 
taking courses almost randomly, students rarely know 
one another and therefore do not serve as resources 
for one another; mastering “literacy” is therefore an 
individual responsibility, with the teacher as the sole 
authority, rather than a collective and social activity 
(Worthen, 1997). 

This approach takes place not only in classes 
identified as remedial; it also emerges as “hidden 
remediation” in college-level classes that become 
remedial if the majority of students are not ready for 
what the instructor considers “college-level work” 
(Grubb and Associates, 1999, Ch. 5). As a result, the 
amount of remediation in most community colleges 
almost surely exceeds the count of official remedial 

COuTSSS. 

Computer programs in remedial classes invariably 
involve drills. Following a rigid progression through 
topics, students move to the next level only when they 
have passed a short “test” on one subject. Often, 
students work on these programs in large labs 
overseen by a “manager” who typically has neither the 
time nor the training for instruction: students who get 
stuck have to go back in the computer program to try 
to work out the problem. There is no teaching in the 
conventional sense of the term. 

Conventional “skills and drills” approaches violate 
all the maxims for good teaching in adult education 
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(Grubb and Kalman, 1994). Although there are some : 
success stories, it is foolish to think that students who 
have never learned to read for meaning, or who have 
no real understanding of numerals, can suddenly learn 
quickly from another round of drills. 

Student-centered teaching. Substantial numbers of 
community college instructors have developed 
student-centered approaches to teaching (Grubb and 
Associates, 1999, Ch. 5). These instructors probe the 
interests of their students and their purposes for 
attending college, and mold reading and writing to 
these interests. These instructors also foster work in 
groups rather than individual drills. But because a 
student-centered class proceeds in different ways 
depending on the backgrounds and interests of 
students, the topics these classes cover are 
idiosyncratic. This creates problems for evaluation, 
because the outcomes are not necessarily well 
defined; in fact, they are partly student- defined. 

Departments with coherent philosophies. In a few 
institutions, developmental studies departments have 
developed a coherent teaching philosophy, drawing on 
constructivist and whole language practices. They 
have then codified their approaches in manuals and 
examples, and then in staff development for new 
instructors (including the many part-time instructors). 
The distinct advantage of department-wide 
approaches over individual and idiosyncratic efforts is 
that they assure that all developmental courses within 
a college are reformed, rather than just a few. 

Learning communities. A fourth major approach to 
developmental education in community colleges is the 
use of learning communities (LCs). Typically, a “lead” 
course — an occupational or central academic course — 
is matched with English and/or a math course. One 
institution matched a biology course with supportive 
math and English courses, which in turn modified their 
content to provide the kinds of academic 
competencies necessary in biology. At LaGuardia 
Community College in New York, all programs for 
welfare recipients are taught in learning communities. 

LCs have many benefits. Most obviously, students 
progress in subjects that they care about rather than 
being in classes with no apparent relation to their 
goals. Combining classes allows instructors to 
contextualize their teaching and enables students to 
get to know one another much better than most 
community college students do. Moreover, in LCs, 
instructors can create communities of like-minded 
teachers. 

Clearly, remedial education can vary enormously; 
even within one institution — the community college — 
remediation is not just one thing. 

Multiple Approaches to Evaluation 

But does remedial/developmental education 
work? Do any of the approaches enable students to 
make further progress in their education or move out 
into the world of employment? The evidence is 



sparse, and partly it is for lack of trying: most states 
and most colleges have not yet evaluated their 
remediation programs. 

One common form of evaluation examines 
completion rates in developmental courses. But this 
information provides no evidence about whether 
remedial programs have helped individuals get along 
with their lives. Another common way of evaluating 
remedial efforts has been the comparison of pre-tests 
and post-tests, usually on some test of basic skills. 
The Learning Assessment and Retention Consortium 
(LARC) of the California community colleges used to 
publish such figures (e.g., LARC, 1989a and b). Such 
results are almost useless, for a number of reasons. 
Most obviously, pre-test/post-test comparisons are 
available only for students who stay with a course 
until the end; if weaker students drop out, then the 
test increases will badly overstate the results for the 
average or random student. In addition, without 
knowing about the backgrounds of students in 
different colleges, comparisons are impossible to 
interpret. 

Moreover, an increase in test scores may still not 
lead to the completion of meaningful degrees or other 
positive outcomes. Finally, these statistical results say 
nothing about why test scores are what they are, why 
large numbers of students fail to complete courses, 
or whether some approaches are better than others. 
Such figures do not provide any guide about what to 
do next. 

One approach to evaluation fixes some of these 
problems but not others. Miami-Dade Community 
College has calculated completion rates for students 
who are judged “below standard” in one, two, or 
three subjects, and who have successfully completed 
all appropriate remedial courses compared to those 
who have not. Students with three deficiencies had a 
much harder time than students with one deficiency, 
and even students who took the full complement of 
needed remedial courses graduated at much lower 
rates than those who entered needing no 
remediation. Clearly, remedial courses help, but they 
do not eliminate the gap between students with and 
without some need for remedial education. 

Despite weaknesses — failure to control for 
variation in academic achievement and other 
characteristics (like family background) among 
groups, for example — these results clarify that the 
amount of remedial education completed matters a 
great deal, and the outcome measure is one of 
intrinsic value. However, they provide no clue about 
why so many students fail to complete remedial 
courses or what about them attracts or repels 
students. Further, they do not investigate what these 
courses are like and whether some of them are more 
effective than others. 

A final issue important to remediation is what I 
call the “assignment problem.” Students are assigned 
to remediation based on an assessment of some 
sort — usually a basic skills test, sometimes with a 



writing sample, sometimes with some counseling. 
Some students assigned to remediation may not 
need it, for example, those needing only brush up. 

But what happens to those students who scored just 
above the cut-off point and did not enroll in remedial 
programs? It is quite possible that these students 
would benefit from remediation. Any procedure runs 
the risk of errors — assigning students to remediation 
who do not need it (Type I errors) — or allowing 
students who need some kind of remedial work to 
progress to college-level work (Type II errors). Both 
errors may reduce completion rates, and Type II 
errors have the added cost of putting unprepared 
students in regular classes. 

These assignment issues are at the heart of 
debates about community colleges: do they advance 
students who otherwise would have no access to 
postsecondary education, or do they “cool out” 
students who otherwise would go further in four-year 
colleges? A particular incident illustrates this issue. 
California instituted a process, known as 
“matriculation,” to help place entering students 
correctly in regular and remedial/developmental 
classes. However, some colleges used tests that had 
not been appropriately validated, preventing many 
individuals from enrolling, and used test results by 
themselves when they were to be advisory only. The 
Mexican-American Legal Defense and Education 
Fund (MALDEF) successfully sued a college on equal 
protection grounds, claiming that many Mexican- 
American students were incorrectly assigned to 
remedial education. In response, the state imposed 
regulations requiring that any prerequisites for any 
courses be justified through a validation study, a 
burdensome procedure that has all but eliminated 
prerequisites in California community colleges — and 
enrollment in remedial courses became voluntary. 
Thus, there may be fewer Type I errors but more Type 
II errors, i.e., students who need more remedial 
education than they get. 

With evaluation of remedial education still in its 
infancy, no one knows much about what works and 
what does not. The little evidence indicates that 
completion rates in remedial courses are low, that the 
amount of remediation matters to important 
outcomes like persistence in and completion of 
college programs, and that learning communities are 
probably more effective than stand-alone classes. 
Observational evidence indicates how much remedial 
courses vary, though my interpretation — that many of 
them provide virtually no possibility for significant 
learning — might not be widely accepted and might 
not even be correct. There is no particular reason to 
think that the remedial courses at Miami-Dade are 
particularly innovative, yet they have substantial 
effects on graduation and retention. But, aside from 
the possible recommendation to teach all remediation 
in LC formats, there is not much evidence to suggest 
how to improve the state of remedial/developmental 
education. 



An Eclectic Approach to Evaluation 

In this vacuum, it is not helpful to recommend one 
particular approach to evaluation over others. The 
orthodoxy in the evaluation literature — random- 
assignment studies to rule out as many selection and 
self-selection effects as possible — is no more useful 
than the most basic pre-test/post-test designs. Too 
many dimensions of remedial education are poorly 
understood; investigating them requires several 
different methods. 

Evaluation is useful on at least two levels. One is 
the program level, where information about a particular 
course and a specific instructor — completion rates, 
assessments of academic progress, subsequent 
progress through the college, and peer observation — 
could diagnose what is going well and badly. To be 
useful to instructors, such evaluations cannot use 
comparison groups, follow students over long periods 
of time, or introduce assessments unrelated to normal 
teaching. A second level includes formal evaluations 
carried out at the institutional level or the state or 
national level. These can be more complex, with 
control or comparison groups, and can follow students 
over longer periods of time; their purpose is not to 
improve the practice of specific instructors, but to 
assess institutional and state policy, the overall effects 
of remediation, and the effectiveness of different 
approaches. 

I recommend a number of different approaches, 
each of which has the potential to illuminate a different 
aspect of this difficult problem: 

1. Dropout rates from remedial courses need more 
investigation. While it is plausible that dreary teaching 
is the reason, the difficult lives of many community 
college students — including financial, transportation, 
childcare and other family problems, and the pervasive 
indecision of experimenting and uncommitted 
students — play important roles. Complex combinations 
of reasons are responsible, and even students 
themselves cannot articulate why they stay with or 
leave a particular program. As one student commented 
on his leaving the community college, 

It was not even a decision. I just didn't go. 
Sometimes you decide on certain things. It was not 
a decision at all. Just like you go home, tired from 
work, you don't decide about “Qh, I'm just going to 
go to sleep now. ” You just doze off and go to 
sleep. It wasn't a plan. That's the way [dropping] 
the class was: it wasn't a plan. 

A combination of qualitative, interview-based studies 
and quantitative studies might begin to provide 
evidence for improving remedial courses. 

2. More systematic collection of outcome 
measures could build to a better understanding of 
what remedial courses can achieve. But outcome 
measures need to include more than test scores of 
basic skills. Persistence in college and completion of 
degrees are obvious measures, because completion is 
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particularly important to the economic benefits of 
community colleges. Some teachers use writing 
portfolios as measures of success. Other measures 
emerge from a college's intentions for remedial 
programs — completion of occupational programs may 
be the most valuable outcomes in some cases. Some 
students have political or familial goals, and qualitative 
studies can clarify these goals and the contribution of 
courses toward achieving them. 

3. It is important in institution- or state- or 
national-level studies, to have comparison or control 
groups. But while it might be possible to design a 
random-assignment study under certain conditions — 
for example, comparing the effects of learning 
communities to conventional formats — it would not 
be ethical or feasible to compare the effects of 
remediation to' its absence through random 
assignment. The only feasible place to construct 
comparison groups is where some students are 
thought to need remediation but do not take such 
classes. 

4. No evaluation should fail to understand the 
program it is evaluating; classroom practices in 
remedial courses must be observed and described. 
Conventional “black box” evaluations, in which the 
nature of the program being evaluated is never 
described, should be replaced with a “Pandora's box” 
approach that clarifies both the triumphs and the 
troubles of classroom practices. Otherwise it is difficult 
to know what might have generated a particular set of 
outcomes, and therefore what might be changed. 

5. If evaluation is to have any influence on 
classroom practice, it needs to compare different 
approaches to teaching. While I am skeptical about 
“skills and drills,” we have observed drill-oriented 
remedial classes where students seemed to be 
attentive and engaged, possibly because the class was 
followed by an occupational class where the academic 
material would be applied. Some teachers following 
behaviorist approaches develop special exercises, or a 
rapport with students, that overcome the limits of drill, 
and some students — particularly ESL students and 
older students with clear and passionate goals — are 
able to learn from even the most dreary teaching. 

Some of these successes may be replicable and 
others may not, but understanding them better is a 
necessary first step to improving the quality of 
instruction. 

6. The “assignment” problem needs to be better 
understood. Understanding this issue depends first on 
ascertaining whether remedial programs themselves 
are effective: if they are ineffective, then every student 
in them is misassigned. However, the question is 
whether some students who might benefit do not 
attend remedial programs — either because the 
assignment test fails to identify those in need of 
remediation, or because enrolling in such courses is 
voluntary. Examining this problem requires looking at 
the subsequent experiences of: (a) students judged in 
need of remediation who did not enroll in such 



courses, and (b) those who barely passed the 
assignment test, compared both to those who enrolled 
in remediation and those who clearly do not need 
remediation. Finally, some consideration of alternative 
assignment procedures is appropriate — either different 
basic skills tests, or procedures that incorporate other 
information and counseling as well as testing. 

The expansion of postsecondary education since 
the 1960s, especially the growth of open-access 
community colleges, has provided opportunities for 
some students where none existed before, and the 
dedication of many colleges and most instructors to 
their non-traditional students is unmistakable. But 
dedication and student-centeredness, while necessary, 
may not be sufficient, so a program of evaluation and 
improvement is central to improving the performance 
of students.*? 0 
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